Encoding worker infra/auth failures: retry instead of terminal-failing the job - #972

Why

Prod alert burst (2026-08-31): a paying customer's job (6452888e) was terminally failed — its render worker error was:

Encoding job 6452888e failed: Failed to retrieve https://metadata.google.internal/.../service-accounts/default/ … Compute Engine Metadata server unavailable … SSLCertVerificationError

Root cause: the c4d primary pair was in a zonal stockout (503 on start), so the render fell back to encoding-worker-fallback-c4a. That fallback VM booted and accepted the job but then couldn't reach the GCE metadata server to fetch its service-account token, so it couldn't touch GCS and died mid-encode. This surfaced as a bare RuntimeError, hit render_video_worker's generic except Exception → fail_job, and lost the job with no retry — even though the exact same work succeeds on any healthy worker.

The same alert burst also included the daily E2E happy-path failure (the encoder being offline made the preview step flaky) and a benign match-judge AI call failed; returning no-suggestion red-error page. All three are addressed here.

What

1. Encoding infra/auth failures are now recoverable (primary fix)

• encoding_errors.py: is_worker_infra_error() classifier (metadata / SSL-cert / missing-credentials markers, deliberately specific so a real ffmpeg/input error never matches) + new EncodingWorkerInfraError — a subclass of EncodingWorkerStartError so it flows through the existing park-for-auto-retry path with zero changes to the render worker's except (and the final-encode Cloud Run Job retries too).
• encoding_service.wait_for_completion(): on a worker-reported infra/auth failure, demote the broken worker + invalidate the URL cache, then raise the recoverable typed error instead of RuntimeError.
• encoding_worker_manager.demote_active_worker(): records a capacity-state cooldown for the serving fallback's machine_type@zone and clears active_override, so the retry re-selects a different, healthy VM. Scoped to fallbacks only — it never family-demotes the fast c4d primary on a rare blip.

2. match-judge noise — the two graceful-degradation logger.exception calls become logger.warning(exc_info=True), so a transient Vertex/Gemini blip that safely returns no-suggestion stops paging as a red "new error pattern".

3. Happy-path E2E robustness — tolerate slow encoder cold-starts at the preview step: scope the alert check to the modal (kills the spurious empty-[role=alert] "Preview error:" from run #151) and recover the "Proceed to Instrumental" button via reload+reopen. The final assertion stays authoritative, so genuinely broken generation still fails the smoke test.

Testing

• New unit coverage: classifier positive/negative + subclass (test_encoding_errors.py), demote_active_worker (fallback demote+clear, primary no-op, never-raises), wait_for_completion infra classification (+ demote/invalidate, and no-manager path), and render worker parks (not fails) on EncodingWorkerInfraError.
• python -m pytest across the encoding / worker / match-judge suites: 177 passed locally.

Not in this PR (infra, needs Andrew / GCP write)

• Fault A: encoding-worker-a (c4d, us-central1-c) is chronically 503 on start (zonal stockout) — that's why the "Encoder offline" footer showed in the E2E. This change makes the fallback path robust to it, but the underlying c4d capacity/zone choice is an infra decision (secret/VM change; my GCP access is read-only). Worth deciding whether to repin the primary family/zone or promote a fallback.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Detects infrastructure and authentication failures on encoding workers.
  • Demotes affected workers and retries jobs using alternate capacity when available.

• Bug Fixes

  • Prevents recoverable worker failures from becoming terminal job errors.
  • Reduces misleading error-level logging for gracefully handled match-judge failures.
  • Improves preview-flow recovery by waiting for actions to become enabled.

• Documentation

  • Clarified that retries prefer, but do not guarantee, a different worker.

• Chores

  • Updated the application version to 0.217.0.